The invention relates to a method for generating a composite image of an object composed of multiple sub-images, wherein multiple adjacent sub-images of the object are captured; and said sub-images are combined together by means of an image processing operation to form a coherent composite image.
When biological samples are analyzed, these samples are exposed to a suitable light and are photographed by means of a camera. By suitably marking with biological markers, the biological samples can be examined by, for example, a fluorescent analysis or with colorimetric techniques. Additional examinations can be performed with simple reflected light or transmitted light techniques. For accurate optical scanning of the sample, microscope optical systems are used in order to identify even small biological structures.
If large samples or so-called micro-arrays with a plurality of objects, probes or biochips are examined, there is the problem that a large area has to be photographed with a high resolution. In order to solve this problem, it is known to scan large areas. In so doing, the area is photographed line by line; and the lines are combined to form a composite image. Such a method is known from the U.S. Pat. No. 7,978,874B1.
Compared with such methods, image detectors offer the advantage of a larger optical variability. However, in order to create a composite image, multiple sub-images that are captured by the image detector have to be merged together to form a composite image. In the case of such an image processing operation, also called stitching, the overlapping regions of the sub-images are examined as to whether they match up. For example, an optical object is found in both of the adjacent sub-images, and the position of the sub-images in relation to each other is determined by means of the position of said optical object in the sub-images, so that the sub-images can be stitched together in the correct position. This technique is known, for example, from U.S. Pat. No. 7,778,485B1.
Such a blending together of sub-images to form a composite image can result in inaccuracies if there is inadequate information in the overlapping region of two sub-images, so the position of the sub-images in relation to each other can be determined only inadequately.
In order to solve this problem, it is known, for example, from the WO98/44446 to move a sample substrate with such a high mechanical precision that the adjacent and non-overlapping sub-images of the object can be stitched together with an accuracy of a few μm without image processing. However, this method requires a motion device with which the sample can be moved under the objective lens with high precision.
The object of the present invention is to provide a simple and reliable method for generating a composite image from multiple adjacent sub-images.
This object is achieved by means of a method of the genre described in the introductory part of the specification, where the stitching of the sub-images together to form a composite image in the image processing operation is performed using an optical pattern that is generated by means of a pattern means. The pattern provides adequate optical information, so that the position of the sub-images in relation to each other can be reliably determined by means of the properties of the pattern, such as its location in the sub-images.
In this case it is advantageous if the sub-images at least partially overlap each other, so that the pattern is imaged in the overlapping region of both sub-images or in the pattern images that overlap each other when the pattern is not directly imaged in the sub-image. However, an overlapping of the sub-images is not absolutely necessary because the optical pattern can also be configured in such a way that immediately adjacent sub-images can be fixed in their position to each other by means of the pattern.
The sub-images depict a part of the object. The composite image depicts a larger part of the object or the whole object and is expediently composed of at least two sub-images, in particular, of at least a one-dimensional chain of K sub-images, where K≥3; and, furthermore, in particular, is composed of a two-dimensional arrangement of N×M sub-images, where N≥2 and M≥2. The sub-images can abut each other or overlap each other. The sub-images are at least one-dimensional images with k pixels, where k≥2, expediently two-dimensional images with a two-dimensional pixel arrangement, expediently an arrangement of n×m pixels, where n≥2 and m≥2. Correspondingly an image detector of an image capturing device for capturing sub-images comprises at least k adjacent detector elements that are arranged, in particular, at least n×m in a two-dimensional grid.
An algorithm in the image processing operation expediently uses the optical pattern to stitch together the sub-images. The image processing method can include one or more of the following steps:
At least two images that at least partially contain the optical pattern are examined for pattern elements of the pattern. Optionally the pattern elements can be recognized as such by means of a pattern recognition device. The pattern elements that are found in the two images are compared with each other for agreement. The agreement can be examined in the form of parameters: size and/or color. If the agreement in one or more parameters exceeds a predefined degree, then the two pattern elements are assigned to each other. Complementary pattern elements are examined for their location in relation to each other. The location can comprise the position and the orientation in the images. The location of the pattern elements can be used to relate the two images in their position to each other. The two images are blended by means of this relationship together to form the composite image. In the composite image two complementary pattern elements are congruently superposed one on top of the other.
The pattern means is arranged expediently outside the object. The pattern means generates an additional pattern for imaging the object. Therefore, the pattern is not a pattern that is inherent in the object. The pattern means is expediently independent of the object. As an alternative, the pattern means may be a component of the object, for example, a pattern means that is mounted on a sample carrier. Even pattern elements, such as particles, in a sample element, for example an adhesive, are possible and advantageous. In general, it is advantageous, if the pattern means is arranged in a region or is imaged as a pattern in a region that is free of the object to be examined or free of a region of the object that is to be examined so the pattern does not disturb the contour of the object.
The optical pattern can consist of one or more pattern elements. The pattern can be a predefined pattern that is generated in such a way that its geometry is defined by the pattern means, for example, dots and/or lines. However, it is also possible to generate the pattern in such a way that it is undefined, so that it is not predetermined. The image processing unit comprises one or more algorithms for the pattern recognition; this algorithm expediently recognizes the pattern as such in the sub-image or in the pattern image.
The optical pattern can be imaged totally or partially in the sub-images, expediently in an overlapping region of two adjacent sub-images. However, it is generally not necessary that the pattern be arranged—if desired, only partially—in the overlapping region. If the pattern or a portion of it is arranged in a sub-image, one property of the pattern can be registered from one edge of the pattern (for example, a brightness profile).
If the optical pattern is imaged totally or partially in the sub-images, then the image processing unit can recognize the optical pattern—or more specifically a pattern element directly from the sub-images—and can stitch the sub-images together in the correct position. However, the imaging of the optical pattern in the sub-images has the drawback that the optical pattern hides the image contents of the sub-images and, as a result, can reduce the image content of the sub-images. In order to eliminate this problem, it is advantageous to image the optical pattern in one or more pattern images that expediently have a defined relative location in relation to the sub-images. It is advantageous if the pattern images are different from the sub-images in such a way that the optical pattern is imaged only in the pattern images, and the sub-images don't have a pattern.
It is advantageous if each sub-image is assigned a pattern image. In this case different sub-images are assigned different pattern images. The image processing unit determines the position of the sub-images in relation to each other by means of the location of one pattern element in two images, i.e. two sub-images or two pattern images. The position of the sub-images in relation to each other can also be determined indirectly by means of the location of the pattern images in relation to each other.
In order to capture the sub-images, the object is moved expediently in relation to an image capturing device, such as a camera or an objective lens. As a result, the image capturing device can scan the object sub-image by sub-image. Advantageously the pattern is moved along with the object. Thus, the pattern moves in the same way relative to the image capturing device as the object itself. At the same time it is advantageous if the pattern, in its movement, is coupled completely to the movement of the object, so that no relative movement between the pattern and the object takes place. Since the movement is only a relative movement, it is also possible to hold the object and the pattern in a state of rest and to move the image capturing device. Advantageously the pattern means is moved along with the object holder and the object. In particular, the pattern means is moved automatically with said object holder. In this case an automatic movement can be generated by means of a rigid connection between the pattern means and the object holder.
One advantageous embodiment of the invention provides that the object holder is a sample stage; and the object is a sample, in particular a biological sample, that is arranged on the sample stage. A biological sample includes a biological material, in particular, cells, micro-organisms, bacteria and/or the like. The sample is expediently a two-dimensional, flat sample, which can exhibit one or more sample elements, for example, microchips, an arrangement of multiple sample vessels and/or the like. Advantageously the sample comprises a sample substance that is placed on or in the sample holder. The sample can be covered by a cover, expediently a cover glass. Advantageously the sample holder and, in particular, also the cover are transparent in the wavelength of the object image. A suitable material for the sample holder is glass.
An additional advantageous embodiment of the invention provides that the method is a microscopy method; and that the sub-images are captured by means of the optical system of a microscope. The optical system of the microscope expediently has a magnification of at least a factor of 5 and, in particular, a magnification of at least a factor of 20.
In order to image the optical pattern on an image detector, there are a number of possibilities that depend on multiple parameters. These parameters are described in detail below and can be combined in any way.
The first parameter is the execution of the pattern as a material pattern or a non-material pattern. Hence, the parameter has two possibilities. The pattern can be a material pattern, for example, an item, such as a glass scale, which is securely connected to the object holder, and/or an item, which is securely connected to the object, for example, a sample. Some examples may include particles in the sample, for example, in an adhesive, or a coating of an element of the object. In contrast, a non-material pattern, for example, a radiation pattern, which can be executed in the form of an imaging, is particularly advantageous. The pattern is thrown by, for example, the pattern means, as a radiation pattern onto a pattern carrier, for example, the object. The term radiation pattern also includes a shadow pattern. A projection of a material pattern onto the pattern carrier is advantageous. Expediently the pattern means is configured in such a way that the optical pattern can be switched off; that is, the optical pattern can be switched on and off.
A second parameter is the wavelength, or more specifically the radiation spectrum, in which the pattern is captured on an image detector. The term wavelength is used below not only for a sharp wavelength, but also for a spectrum of wavelengths. Images of the pattern or a portion of it can be captured in the wavelength of the object image or in a wavelength of the pattern image. Hence, these parameters also have two possibilities: that is, a dual parameter. The wavelength of the object image is a wavelength or rather an optical spectrum, in which the sub-images are captured and in which an object image detector that captures the images is sensitive. The wavelength of the image can be in the visible spectral range, can be a fluorescent wavelength or any other suitable spectral range.
A wavelength of the pattern image is different from the wavelength of the object image. Said wavelength of the pattern image lies outside the wavelength of the object image, or more specifically outside the wavelength spectrum of the object image. Said wavelength of the pattern image is such a wavelength, or rather spectral range, in which the sub-images are not captured, and, thus, to which the object image detector is insensitive or which is filtered out of the optical path of the object image. In the course of capturing sub-images, the wavelength of the pattern is, for example, filtered out of the optical path so that the sub-images are not captured in this wavelength. In an additional possibility the wavelength of the pattern is invisible to the object image detector. Hence, said object image detector is insensitive in this wavelength.
An additional parameter constitutes the excitation wavelength and the emission wavelength of the pattern. Typically the emission wavelength of the pattern is included in the excitation wavelength spectrum of the pattern. The pattern is generated, for example, with blue light, emits with blue light, and is captured in the blue spectral range. However, there is also the possibility of configuring the pattern in such a way that its emission wavelength is not included in the excitation wavelength spectrum, for example, in the case of fluorescence. In this case a highly energetic radiation is incident on a pattern carrier, which emits the pattern with low energetic radiation, in which the pattern is then captured. Hence, this parameter is also a dual parameter, and it therefore has two possibilities.
An additional dual parameter has the two possibilities that the pattern is already incident on a pattern carrier as a pattern, for example, the object, or said pattern is first formed by the pattern carrier; that is, the pattern is inherent in the pattern carrier. In this case the pattern carrier is, for example, uniformly exposed; and said pattern carrier generates the pattern in its emission, for example, in that only one pattern region scatters, reflects or fluoresces.
Another dual parameter consists of whether the pattern is imaged in the sub-image, thus in the imaging of the object, which is used for generating the composite image, or whether the pattern is imaged in a pattern image, which is in addition to the object image. In this case the pattern is not imaged in the sub-image, so that the object information in the sub-image is maintained to the maximum extent.
When pattern images are captured, it is advantageous if the pattern images are arranged in a predefined position in relation to the sub-images. The position of the pattern images in relation to each other can be determined by means of an evaluation of the pattern in the pattern images. The position of the sub-images in relation to each other can be determined by means of the known position of the object images in relation to the pattern images, so that these sub-images can be stitched together to form the composite image.
The pattern image can comprise an imaging of the object. However, it is not used as a sub-image. For example, the object is imaged together with the pattern on an object image detector; and a pattern image is captured. The position of the pattern image, particularly in relation to a predefined reference point, is determined by means of the pattern. The sub-image is captured beforehand or afterwards without the pattern, so that there is the possibility of removing the pattern from the imaging and then capturing the sub-image or first capturing the sub-image and then making the pattern visible in the imaging on the detector. Expediently the object holder is held in the same position, for example, relative to the reference point, in both images.
An additional parameter is the place of a non-material pattern. The pattern becomes visible at a pattern carrier. This pattern carrier can be the object or an item that is different from the object, for example, at the object holder or at a different item.
Furthermore, it is advantageous if the pattern is made visible on a pattern carrier that is transparent to a wavelength of the object image. The pattern carrier can be transilluminated in the object image wavelength in which the sub-images were captured, whereas the pattern can be emitted by the pattern carrier. In this case the pattern carrier is expediently opaque to a wavelength of the pattern image. The pattern carrier can be a glass, for example, a sample carrier and/or a cover glass for covering a sample. An adhesive that glues multiple elements together is also possible. The pattern carrier water is similarly possible and advantageous, for example, as a component of the object, while other components of the object can also be used as the pattern carrier at the same time. Glass can be used as such as the pattern carrier or can comprise a stain that is transparent to the wavelength of the object image and is opaque to the wavelength of the pattern image.
Another advantageous embodiment of the invention provides that the pattern is a laser pattern. A predefined pattern is especially easy to generate with a laser. The laser pattern is made visible in an advantageous way on a pattern carrier that is expediently not the object or a part of it. The laser can radiate, just like the laser pattern, in the wavelength of the object image or in a separate wavelength of the pattern image. Similarly it is possible that the laser pattern is excited by the laser and radiates back in a wavelength that is different from the laser.
Furthermore, it is advantageous if the pattern is an interference pattern. This interference pattern can be generated in a pattern carrier, which can be, for example, a cover glass of a sample, and which is transparent to the wavelength of the object image. In this case the interference pattern can be imaged as such on an object image detector or a pattern image detector.
The pattern can be generated, with the same advantage, as a standing wave, in particular in a sample element, i.e. an element associated with the sample: for example, the sample holder or a part of it, such as a sample cover glass.
The generation of the pattern as a speckle pattern is similarly possible and advantageous. This speckle pattern is also generated advantageously in a sample element.
An additional dual parameter is the chronological sequence of the capturing of the sub-images and the pattern images. The capturing can occur simultaneously or in succession. During image capturing in succession, a sub-image can be generated first, and then beforehand or afterwards the pattern image, assigned to the sub-image, can be generated. The sub-image and the pattern image can be imaged on the same object image detector, or the sub-image is imaged on the object image detector, and the pattern image is imaged on a pattern image detector.
Another dual parameter consists of imaging the pattern on either an object image detector or a pattern image detector, which is separate from the object image detector. In the event that two detectors are used, these detectors can be sensitive in the same wavelength of the object image or in different spectral ranges: for example, the object image detector in the wavelength of the object image and the pattern image detector in the wavelength of the pattern image.
An additional parameter is the choice of the objective lens that captures the pattern. The pattern can be imaged with an object objective lens on a detector, and this object objective lens also images the object on the object image detector. Another possibility consists of imaging the pattern with a pattern objective lens, which is expediently present in addition to the object objective lens, and by means of which the object is not imaged. The parameters of both detectors and the two objective lenses can be combined with each other in any way. Thus, an optical path can be guided from two objective lenses to one detector, or from one objective lens to two detectors, or from two objective lenses to two detectors, or from one objective lens, which images the object and the pattern, to a single detector.
An additional parameter consists of the relative position of the pattern in relation to the object. The pattern can be made visible on or in the object, expediently in the plane of the object image. Another possibility consists of arranging the pattern in the object image, but before or after the plane of the object image. To this end, it is advantageous if the pattern is arranged outside the object plane of the sub-images; i.e. outside the plane on which the optical system focuses on the object image detector for sharp imaging of the object. In this case the optical system can focus initially on the pattern and then on the object or vice versa. The images, i.e. the sub-images and the pattern images, can be captured as a function of the focusing. A third possibility for the parameters consists of arranging the pattern next to the object and outside the object image.
Another parameter can include whether the radiation that generates the optical pattern is incident on the object, or whether the pattern goes through the object. If the radiation is incident on the object, then the radiation can be incident on the object by means of the objective lens of the object. That is, the radiation can be coupled into an image optical path of the object image. As an alternative or in addition, the radiation can be transmitted from externally past the objective lens of the object onto the object. If the pattern radiates in the transmitted light technique, then said pattern is passed through the object or the object holder.
The choice of a suitable pattern means offers an additional parameter. The pattern means can be a transmission element, i.e. such an element that is illuminated by a light source and allows only a portion of the radiation to pass as the light pattern or shadow pattern. Another embodiment uses a self-radiating pattern, for example, an LED, which already forms a pattern by means of its non-uniform emission, so that in one advantageous embodiment of the invention the pattern can be an imaging of a pattern light source.
The aforementioned parameters can be combined in any way so that an advantageous generation of the pattern is achieved for the respective application. Individual advantageous exemplary embodiments are described in conjunction with the figures. However, due to the plethora of possible and advantageous embodiments corresponding to the combination of the aforementioned parameters, it is not possible to describe all of the advantageous embodiments. Correspondingly only a small selection of advantageous embodiments can be presented. However, other combinations of the parameters may be advantageous in other applications.
In addition, the invention relates to a device for generating a composite image composed of multiple sub-images. The device comprises an object image detector, an optical system for imaging an object on the object image detector, and an image processing unit for stitching multiple adjacent and, in particular, partially overlapping sub-images of the object together to form a coherent composite image.
It is proposed that the device exhibits a pattern means for generating an optical pattern according to the invention. In this case the image processing unit is preconfigured to stitch the sub-images together to form the composite image by means of the pattern. As described above, the relative position of the sub-images in relation to each other can be recognized by means of the pattern, and the sub-images can be stitched together in the correct position to form the composite image.
The device comprises advantageously one or more of the above described device elements, such as detectors, objective lenses, sample elements, pattern carriers and the like.
Furthermore, the device advantageously comprises a control unit that is preconfigured to carry out the method according to the invention and one or more of the above described details of the method.
The optical system expediently comprises an object optical system, which is transmissive in a wavelength of the object image, and which can also comprise an object objective lens. Furthermore, the optical system comprises advantageously a pattern optical system, which is transmissive in a wavelength of the pattern image and which can also comprises a pattern objective lens. Expediently, an object optical system is less than 50% transparent in a wavelength of the pattern image, in particular less than 10%.
The above description of advantageous embodiments of the invention includes numerous parameters of the invention that are described in detail. However, the person skilled in the art will combine these features in a practical way to form logical combinations. In particular, these features can be combined individually and in any suitable combination with the inventive method and the inventive device, according to the independent claims.
The above described properties, features and advantages of the invention as well as the manner in which said properties, features and advantages are achieved, will become clearer and will be better understood in conjunction with the following description of the exemplary embodiments that are explained in detail with reference to the drawings. The purpose of the exemplary embodiments is to explain the invention and not to limit the invention to the combination of features that are disclosed in said exemplary embodiments, not even with respect to the functional features. In addition, the respective suitable features of any one of the exemplary embodiment may also be viewed explicitly in isolation, may be removed from one exemplary embodiment, inserted into another exemplary embodiment in order to complement it, and/or can be combined with any one of the claims.